{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 概述\n",
    "\n",
    "前几节我们尝试使用与房价预测相同的简单神经网络解决手写数字识别问题，但是效果并不理想。原因是手写数字识别的输入是28 × 28的像素值，输出是0-9的数字标签，而线性回归模型无法捕捉二维图像数据中蕴含的复杂信息，如 **图1** 所示。无论是牛顿第二定律任务，还是房价预测任务，输入特征和输出预测值之间的关系均可以使用“直线”刻画（使用线性方程来表达）。但手写数字识别任务的输入像素和输出数字标签之间的关系显然不是线性的，甚至这个关系复杂到我们靠人脑难以直观理解的程度。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/959776f4cd9c4b77b380c7d29f59df1cf47be626cd8b4bd1ac1af2a7d8e3c1cf\" width=\"800\" hegiht=\"\" ></center>\n",
    "<center><br>图1：数字识别任务的输入和输出不是线性关系 </br></center>\n",
    "<br></br>\n",
    "\n",
    "因此，我们需要尝试使用其他更复杂、更强大的网络来构建手写数字识别任务，观察一下训练效果，即将“横纵式”教学法从横向展开，如 **图2** 所示。本节主要介绍两种常见的网络结构：经典的多层全连接神经网络和卷积神经网络。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/0820ed2841474d0a87ef72d05111e37e4b5287473dc344e084596896ee552c5b\" width=\"900\" hegiht=\"\" ></center>\n",
    "<center><br>图2：“横纵式”教学法 — 网络结构优化 </br></center>\n",
    "<br></br>\n",
    "\n",
    "\n",
    "### 数据处理\n",
    "\n",
    "在介绍网络结构前，需要先进行数据处理，代码与上一节保持一致。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  from collections import MutableMapping\n",
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  from collections import Iterable, Mapping\n",
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  from collections import Sized\n"
     ]
    }
   ],
   "source": [
    "#数据处理部分之前的代码，保持不变\n",
    "import os\n",
    "import random\n",
    "import paddle\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "import gzip\n",
    "import json\n",
    "\n",
    "# 定义数据集读取器\n",
    "def load_data(mode='train'):\n",
    "\n",
    "    # 加载数据\n",
    "    datafile = './work/mnist.json.gz'\n",
    "    print('loading mnist dataset from {} ......'.format(datafile))\n",
    "    data = json.load(gzip.open(datafile))\n",
    "    print('mnist dataset load done')\n",
    "\n",
    "    # 读取到的数据区分训练集，验证集，测试集\n",
    "    train_set, val_set, eval_set = data\n",
    "\n",
    "    # 数据集相关参数，图片高度IMG_ROWS, 图片宽度IMG_COLS\n",
    "    IMG_ROWS = 28\n",
    "    IMG_COLS = 28\n",
    "\n",
    "    if mode == 'train':\n",
    "        # 获得训练数据集\n",
    "        imgs, labels = train_set[0], train_set[1]\n",
    "    elif mode == 'valid':\n",
    "        # 获得验证数据集\n",
    "        imgs, labels = val_set[0], val_set[1]\n",
    "    elif mode == 'eval':\n",
    "        # 获得测试数据集\n",
    "        imgs, labels = eval_set[0], eval_set[1]\n",
    "    else:\n",
    "        raise Exception(\"mode can only be one of ['train', 'valid', 'eval']\")\n",
    "\n",
    "    #校验数据\n",
    "    imgs_length = len(imgs)\n",
    "    assert len(imgs) == len(labels), \\\n",
    "          \"length of train_imgs({}) should be the same as train_labels({})\".format(\n",
    "                  len(imgs), len(labels))\n",
    "\n",
    "    # 定义数据集每个数据的序号， 根据序号读取数据\n",
    "    index_list = list(range(imgs_length))\n",
    "    # 读入数据时用到的batchsize\n",
    "    BATCHSIZE = 100\n",
    "    \n",
    "    # 定义数据生成器\n",
    "    def data_generator():\n",
    "        if mode == 'train':\n",
    "            random.shuffle(index_list)\n",
    "        imgs_list = []\n",
    "        labels_list = []\n",
    "        for i in index_list:\n",
    "            img = np.array(imgs[i]).astype('float32')\n",
    "            label = np.array(labels[i]).astype('float32')\n",
    "            # 在使用卷积神经网络结构时，uncomment 下面两行代码\n",
    "            img = np.reshape(imgs[i], [1, IMG_ROWS, IMG_COLS]).astype('float32')\n",
    "            label = np.reshape(labels[i], [1]).astype('float32')\n",
    "            \n",
    "            imgs_list.append(img) \n",
    "            labels_list.append(label)\n",
    "            if len(imgs_list) == BATCHSIZE:\n",
    "                yield np.array(imgs_list), np.array(labels_list)\n",
    "                imgs_list = []\n",
    "                labels_list = []\n",
    "\n",
    "        # 如果剩余数据的数目小于BATCHSIZE，\n",
    "        # 则剩余数据一起构成一个大小为len(imgs_list)的mini-batch\n",
    "        if len(imgs_list) > 0:\n",
    "            yield np.array(imgs_list), np.array(labels_list)\n",
    "\n",
    "    return data_generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 经典的全连接神经网络\n",
    "\n",
    "\n",
    "经典的全连接神经网络来包含四层网络：输入层、两个隐含层和输出层，将手写数字识别任务通过全连接神经网络表示，如 **图3** 所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/2173259df0704335b230ec158be0427677b9c77fd42348a28f2f8adf1ac1c706\" width=\"600\" hegiht=\"\" ></center>\n",
    "<center><br>图3：手写数字识别任务的全连接神经网络结构</br></center>\n",
    "<br></br>\n",
    "\n",
    "* 输入层：将数据输入给神经网络。在该任务中，输入层的尺度为28×28的像素值。\n",
    "* 隐含层：增加网络深度和复杂度，隐含层的节点数是可以调整的，节点数越多，神经网络表示能力越强，参数量也会增加。在该任务中，中间的两个隐含层为10×10的结构，通常隐含层会比输入层的尺寸小，以便对关键信息做抽象，激活函数使用常见的Sigmoid函数。\n",
    "* 输出层：输出网络计算结果，输出层的节点数是固定的。如果是回归问题，节点数量为需要回归的数字数量。如果是分类问题，则是分类标签的数量。在该任务中，模型的输出是回归一个数字，输出层的尺寸为1。\n",
    "\n",
    "------\n",
    "**说明：**\n",
    "\n",
    "隐含层引入非线性激活函数Sigmoid是为了增加神经网络的非线性能力。\n",
    "\n",
    "举例来说，如果一个神经网络采用线性变换，有四个输入$x_1$~$x_4$，一个输出$y$。假设第一层的变换是$z_1=x_1-x_2$和$z_2=x_3+x_4$，第二层的变换是$y=z_1+z_2$，则将两层的变换展开后得到$y=x_1-x_2+x_3+x_4$。也就是说，无论中间累积了多少层线性变换，原始输入和最终输出之间依然是线性关系。\n",
    " \n",
    "------\n",
    "\n",
    "Sigmoid是早期神经网络模型中常见的非线性变换函数，通过如下代码，绘制出Sigmoid的函数曲线。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  if isinstance(obj, collections.Iterator):\n",
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  return list(data) if isinstance(data, collections.MappingView) else data\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAH8lJREFUeJzt3Xl4XPV59vHvM6PNkmzZsrxgS7ZsY2xsMBhkY3DCEggYkpi0IWBICJAUSt6QkpbQF5qWpLR9m6VNmrZOAyUkJSyOIaS44BRwgJAQvIONV5A3LV4kb/KidWae948ZO0KxrcEe6cyM7s916Zoz5xzN3JjRraOz/I65OyIikl1CQQcQEZHUU7mLiGQhlbuISBZSuYuIZCGVu4hIFlK5i4hkIZW7iEgWUrmLiGQhlbuISBbKCeqNy8rKvLKyMqi3FxHJSCtWrNjt7kO6Wy+wcq+srGT58uVBvb2ISEYys23JrKfdMiIiWUjlLiKShVTuIiJZSOUuIpKFVO4iIlmo23I3s0fNrMHM1hxnuZnZv5pZtZmtNrPzUh9TREQ+iGS23H8CzDrB8quB8YmvO4D/OPVYIiJyKro9z93dXzezyhOsci3wmMfv17fYzAaa2WnuviNFGUVEPjB3py0Soz0ao60jRkf091/tEScSi9ERdaIxJxKNEYklpmNONPb75zF3ojGIHZl2J+a/fx7z+HvF3HEnvixx+9JYzHE4uswTua44cxjnVAzs0f/+VFzENBKo7fS8LjHvD8rdzO4gvnXPqFGjUvDWIpJN2iJR9jd3sK+5nX2HO2hq6eBAawcHWyMcaOngUFuE5vYIh9qiHE5Mt7RHaW6P0tIRpS0So7UjSltHvNTTkRkMLynIiHJPmrs/DDwMUFVVpTtzi/QRsZiz62Ardfta2NHUyo798ceGg600Hmxj96F2Gg+2cagtcsLXKcoLU5SfQ3F+DoX5YQpzcxhYmMeIgWEKcuNf+Tmho495OaGjj3nhELnhELk5IXJDRm44RE44/hgOGblhIxwKkRMyQmbkhOOP4ZARNiMUgnBimRmEE8ss8TxkRsjA+P1zMzA6TZv1zj84qSn3eqCi0/PyxDwR6WMOtUV4b9dB3ms4RHXDITY1HGLb3mZq9jbTHnn/lnT//ByGDMhnaP98Jo8YQFlxPmXFeQwszKO0KI+BhbmU9MtlQEEuA/rlUpyfQzjUe+WY6VJR7guAu8xsHnAB0KT97SLZr6U9ylu1+3inrol36ptYU9/E1j3NR5fn5YQYW1bEuCFFfGTiUEaVFlJRWsiIkgKGlxTQvyA3wPTZr9tyN7OngEuBMjOrA74O5AK4+w+BhcA1QDXQDNzWU2FFJDitHVEWb97D4s17WbplD6vrmojE4ntXRw7sx9kjS7ju/HImDB/A+KHFVJQWaks7QMmcLXNjN8sd+FLKEolI2tjR1MKi9Q28uqGB323aTWtHjNywMaV8ILdfPJbplaWcUzGQ0qK8oKNKF4EN+Ssi6Wnf4XYWrtnBc29vZ+mWvQCMKi1kzrRRXDZxKNMrS+mXFw44pXRH5S4iuDtvbt7DY7/bxq827KIj6owbUsQ9Hz2Dq88+jXFDinr1TA85dSp3kT6spT3Kz1fW8dibW3l31yEGFuZyy4WVfHLqSCaPGKBCz2Aqd5E+qLUjyuOLt/HDX29i96F2Jo8YwLevm8Lsc0ZQkKtdLtlA5S7Sh7RFosxbWsvcV6tpONjGReMGM/em8UwfU6qt9CyjchfpI3773m4eeG4Nm3cfZvqYUv71xqnMGDs46FjSQ1TuIlluZ1Mrf//COp5fvYPRgwv58a3TuHTCEG2pZzmVu0iWcneeXlHH3y5YS0fM+coV47nzknHap95HqNxFslBTSwdf+8U7PL96BzPGlvKtT01h9OCioGNJL1K5i2SZ5Vv3cve8t9l5oJV7r5rAnZeM0zAAfZDKXSSL/HTxNr6xYC0jB/bjmTsvZOqoQUFHkoCo3EWyQDTm/MML63n0jS18ZOJQvj/nXI262Mep3EUy3OG2CHfPe4tF6xu4bWYlf/2xSdoNIyp3kUy293A7n3t0Ceu2H+DBayfzuQsrg44kaULlLpKh9h5u5zOPLGFz4yF+dMs0Lps4NOhIkkZU7iIZqHOx/+fnqrj4jCFBR5I0Ewo6gIh8MCp2SYbKXSSDHG6LcPOPVOzSPe2WEckQkWiMLz/1Fht2HuQRFbt0Q1vuIhnA3Xnw+XW8sqGBb8yerIOn0i2Vu0gG+PEbW3nszW3c/uEx3DxjdNBxJAOo3EXS3MvrdvF3L6xj1uTh3H/1mUHHkQyhchdJYzV7mvmLn73N2SNL+N4N5xLSlaeSJJW7SJpqi0S566mVmMHcm86jX57GYZfk6WwZkTT1rV9uZHVdEw/dfD4VpYVBx5EMoy13kTT00tqdPPrGFm69qJKrJg8POo5kIJW7SJqp39/Cvc+s5uyRJdx/zcSg40iGUrmLpBF356vzVxGNOf9+01Tyc7SfXU6Oyl0kjTy1tJY3N+/hax87U/c8lVOichdJEzuaWvh/C9dz0bjBzJlWEXQcyXBJlbuZzTKzjWZWbWb3HWP5KDN71czeMrPVZnZN6qOKZC9352u/WEM05nzzj6dgpvPZ5dR0W+5mFgbmAlcDk4AbzWxSl9X+Gpjv7lOBOcAPUh1UJJs99/Z2XtnQwFevmsCowTrtUU5dMlvu04Fqd9/s7u3APODaLus4MCAxXQJsT11Ekey2+1Ab3/iftZw3aiC3XlQZdBzJEslcxDQSqO30vA64oMs63wBeMrMvA0XAFSlJJ9IHfOuXGzjcFuHb103Rja0lZVJ1QPVG4CfuXg5cA/zUzP7gtc3sDjNbbmbLGxsbU/TWIplrVe1+nl5Rx+c/NIbTh/YPOo5kkWTKvR7ofOi+PDGvsy8A8wHc/U2gACjr+kLu/rC7V7l71ZAhutGA9G3uzt/+z1rKivO567LTg44jWSaZcl8GjDezMWaWR/yA6YIu69QAlwOY2ZnEy12b5iInsGDVdlbW7OcvZ02gf0Fu0HEky3Rb7u4eAe4CXgTWEz8rZq2ZPWhmsxOr3QPcbmargKeAW93deyq0SKZrbo/wjws3cPbIEq47rzzoOJKFkhoV0t0XAgu7zHug0/Q6YGZqo4lkr/94bRM7D7Qy9zNTNUa79AhdoSrSy+r2NfPQ65u59twRnD+6NOg4kqVU7iK97PuL3gPg/87SiI/Sc1TuIr1oU+Mhfr6yjs9eMJoRA/sFHUeymMpdpBd97+V3KcgN838uGxd0FMlyKneRXrJu+wGeX72D22ZWUlacH3QcyXIqd5Fe8t2XN9K/IIc7Pqytdul5KneRXrCyZh+L1jfwpxePpaRQFyxJz1O5i/SCf35pI4OL8rht5pigo0gfoXIX6WHLtu7ljeo9fPHScRTlJ3XdoMgpU7mL9LAfvFpNaVEen7lgdNBRpA9RuYv0oLXbm3h1YyOfn1lJv7xw0HGkD1G5i/Sg/3htE8X5Odx8YWXQUaSPUbmL9JAtuw+z8J0dfGbGKEr66QwZ6V0qd5Ee8tCvN5ETDvGFD+kMGel9KneRHrCzqZWfr6zj+qpyhvYvCDqO9EEqd5Ee8MhvNhNz+NOLdTWqBEPlLpJiTS0dPLm0hk9MOY2K0sKg40gfpXIXSbGfLauhuT3Kn3x4bNBRpA9TuYukUCQa479+t40LxpRy1siSoONIH6ZyF0mhF9fuon5/i86QkcCp3EVS6Ee/3cyo0kIuP3NY0FGkj1O5i6TIWzX7WFmzn9tmVhIOWdBxpI9TuYukyKNvbKV/fg6frqoIOoqIyl0kFXY0tbDwnR3cMK2CYg3rK2lA5S6SAo+9uQ1355aLKoOOIgKo3EVOWWtHlKeW1nDlpOG6aEnShspd5BQ9v3oH+5s7+NxFuhmHpA+Vu8gp+unibYwbUsSFYwcHHUXkKJW7yClYXbefVbX7uXnGaMx0+qOkD5W7yCl4fPE2+uWG+ePzy4OOIvI+KneRk9TU3MFzb2/nk1NHMqBAd1qS9JJUuZvZLDPbaGbVZnbfcda53szWmdlaM3sytTFF0s/TK2ppi8S4eYYOpEr66fZqCzMLA3OBjwJ1wDIzW+Du6zqtMx64H5jp7vvMbGhPBRZJB7GY8/jibVSNHsSkEQOCjiPyB5LZcp8OVLv7ZndvB+YB13ZZ53ZgrrvvA3D3htTGFEkvv63ezdY9zdx8obbaJT0lU+4jgdpOz+sS8zo7AzjDzN4ws8VmNutYL2Rmd5jZcjNb3tjYeHKJRdLA44u3Mbgoj1lnDQ86isgxpeqAag4wHrgUuBH4TzMb2HUld3/Y3avcvWrIkCEpemuR3rXrQCu/2tDAp6sqyM8JBx1H5JiSKfd6oPMwd+WJeZ3VAQvcvcPdtwDvEi97kazz9PJaojFnzjSN/ijpK5lyXwaMN7MxZpYHzAEWdFnnv4lvtWNmZcR302xOYU6RtBCLOU8trWXm6YOpLCsKOo7IcXVb7u4eAe4CXgTWA/Pdfa2ZPWhmsxOrvQjsMbN1wKvAve6+p6dCiwTlN9W7qd/fwo3TRwUdReSEkhp42t0XAgu7zHug07QDf5H4EslaTy6JH0i9cpIOpEp60xWqIklqONDKovUNXHd+OXk5+tGR9KZPqEiSnl5RRzTm3KADqZIBVO4iSYgfSK3hwrGDGTukOOg4It1SuYsk4bfVu6nb18KNF+hAqmQGlbtIEuYtq2FQYS5XTR4WdBSRpKjcRbqx+1AbL6/bxR+fV64rUiVjqNxFuvGLlfV0RHUgVTKLyl3kBNydectqOG/UQM4Y1j/oOCJJU7mLnMCKbfvY1HiYOdN0IFUyi8pd5ATmLaulKC/Mx6acFnQUkQ9E5S5yHAdaO3hh9Q5mnzuCovykRuoQSRsqd5Hj+J9V22npiHKDdslIBlK5ixzH/GW1TBzen3PKS4KOIvKBqdxFjmHd9gOsqmvihmkVmFnQcUQ+MJW7yDHMX15LXjjEJ8/tertgkcygchfporUjyrMr65h11nAGFeUFHUfkpKjcRbr43zU7OdAa0T1SJaOp3EW6mLeshlGlhcwYOzjoKCInTeUu0smW3YdZvHkvN0yrIBTSgVTJXCp3kU7mL68lHDKuO7886Cgip0TlLpLQEY3xzIo6LpswlGEDCoKOI3JKVO4iCa9uaKDxYJsOpEpWULmLJPxsWS1D++dz6YQhQUcROWUqdxFgZ1Mrr25s4NNV5eSE9WMhmU+fYhHiB1JjDtdXaZeMZAeVu/R50Zjzs2W1fOj0MkYPLgo6jkhKqNylz3v9vUbq97dw43QN7SvZQ+Uufd5TS2ooK87jo5OGBR1FJGVU7tKn7TrQyq82NHDd+RXk5ejHQbKHPs3Spz29vJZozHVuu2SdpMrdzGaZ2UYzqzaz+06w3qfMzM2sKnURRXpGLOY8tbSWmacPprJMB1Ilu3Rb7mYWBuYCVwOTgBvNbNIx1usP3A0sSXVIkZ7wm+rdOpAqWSuZLffpQLW7b3b3dmAecO0x1vs74FtAawrzifSYJ5dsY3BRHldOGh50FJGUS6bcRwK1nZ7XJeYdZWbnARXu/kIKs4n0mF0HWlm0voFPnV+uA6mSlU75U21mIeC7wD1JrHuHmS03s+WNjY2n+tYiJ+2ppTVEY85N2iUjWSqZcq8HOp9KUJ6Yd0R/4CzgNTPbCswAFhzroKq7P+zuVe5eNWSIBmeSYHREYzy1tIZLzhiiA6mStZIp92XAeDMbY2Z5wBxgwZGF7t7k7mXuXunulcBiYLa7L++RxCKnaNG6Xew60MbNM0YHHUWkx3Rb7u4eAe4CXgTWA/Pdfa2ZPWhms3s6oEiq/XTxNkYO7MdlE4cGHUWkx+Qks5K7LwQWdpn3wHHWvfTUY4n0jOqGg/xu0x7uvWoCYd0jVbKYThOQPuXxxTXkhUPcoCtSJcup3KXPaG6P8PMVdVxz9nDKivODjiPSo1Tu0mc89/Z2DrZFuPlCHUiV7Kdylz7B3XnszW2cedoAzhs1KOg4Ij1O5S59wuLNe1m/4wC3XDgaMx1Ileyncpc+4dE3tlBalMcnp47sfmWRLKByl6y3dfdhFq3fxWcuGEVBbjjoOCK9QuUuWe8nv9tKTsh0Rar0KSp3yWoHWjt4enktn5gygqEDCoKOI9JrVO6S1eYvq+Vwe5TPf2hM0FFEepXKXbJWJBrjx29sZfqYUs4aWRJ0HJFepXKXrPXyul3U72/hC9pqlz5I5S5Zyd155LdbqCjtxxVnDgs6jkivU7lLVlq6ZS8rtu3j9g+P1eiP0iep3CUrzX1tE2XFeVxfpdEfpW9SuUvWWVPfxOvvNvL5D43RRUvSZ6ncJev84LVq+ufn8FldtCR9mMpdssqmxkP8cs1OPnfRaAYU5AYdRyQwKnfJKg/9ehN54RC3zdTpj9K3qdwla2zf38KzK+uZM61Cd1qSPk/lLlnj4dc3A3D7xWMDTiISPJW7ZIX6/S08uaSGT51XTvmgwqDjiARO5S5Z4d9+9R4Af3bF+ICTiKQHlbtkvC27D/P0ijpuumAUIwf2CzqOSFpQuUvG+5dF75IXDvGly04POopI2lC5S0bbuPMgC1Zt59aZlQzprzNkRI5QuUtG++eXNlKcl8Of6gwZkfdRuUvGWlW7n5fW7eL2i8cysDAv6DgiaUXlLhnJ3fn7F9YxuChPt9ATOQaVu2Sk51fvYNnWfXz1qgkU5+cEHUck7ajcJeO0tEf5x4XrmTxigMZrFzmOpMrdzGaZ2UYzqzaz+46x/C/MbJ2ZrTazX5mZxlqVHvPQ65vY3tTK1z8xWXdZEjmObsvdzMLAXOBqYBJwo5lN6rLaW0CVu08BngG+neqgIhAfZuCHv97Ex6ecxvQxpUHHEUlbyWy5Tweq3X2zu7cD84BrO6/g7q+6e3Pi6WKgPLUxReL+ceF63OH+a84MOopIWkum3EcCtZ2e1yXmHc8XgF8ea4GZ3WFmy81seWNjY/IpRYA3N+3h+dU7uPOScRpmQKQbKT2gamafBaqA7xxrubs/7O5V7l41ZMiQVL61ZLmW9ij3P7uaUaWF3HnJuKDjiKS9ZM4hqwc6n5JQnpj3PmZ2BfA14BJ3b0tNPJG47y16l617mnny9gvol6ebXot0J5kt92XAeDMbY2Z5wBxgQecVzGwq8BAw290bUh9T+rJVtft55DebuXH6KC4aVxZ0HJGM0G25u3sEuAt4EVgPzHf3tWb2oJnNTqz2HaAYeNrM3jazBcd5OZEPpD0S4y+fWc3Q/gXcf83EoOOIZIykLu1z94XAwi7zHug0fUWKc4kAMPfVajbuOsijt1YxoCA36DgiGUNXqEraWlPfxA9eq+aT547gIxOHBR1HJKOo3CUtHWzt4EtPrqSsOJ+vf2Jy0HFEMo5GXJK04+7c9+w71O1r4Wd3zGBQkYbzFfmgtOUuaeeJJTW8sHoH91x5BlWVGmJA5GSo3CWtrN3exIPPr+OSM4Zw58W6WEnkZKncJW00NXdw15NvMagwl+9efw4hjfgoctK0z13SQnskxp2Pr6BuXzNP/MkMBhfrZtcip0LlLoFzd/7qF+/w5uY9fO+GczSUr0gKaLeMBG7uq9U8s6KOuy8fzx9N1WjRIqmgcpdAPfd2Pf/00rv80dSRfOWK8UHHEckaKncJzItrd3LP/FVMH1PKNz91NmY6gCqSKip3CcSLa3fypSdWctbIEh65pYr8HA3jK5JKKnfpdUeK/ezyEh77wnQNCCbSA3S2jPSq/12zg7uefIuzy0v4r8+r2EV6ispdeoW786PfbuEfFq5nasVAfqJiF+lRKnfpcR3RGF9fsJYnl9Rw9VnD+e715+pWeSI9TOUuPepAawdfemIlv3lvN1+8dBz3XjlBwwqI9AKVu/SYlTX7uHveW+zY38q3PzWF66dVdP9NIpISKndJuWjM+eGvN/Hdl99l+IAC5t0xQ0P3ivQylbukVN2+Zr769CoWb97Lx6ecxj/80dmU9NOBU5HepnKXlGiLRHnkN1v4t1feI2TGd66bwnXnl+uqU5GAqNzllL1RvZu/eW4NmxsPM2vycP7mE5MYObBf0LFE+jSVu5y0VbX7+d6id3ltYyOjBxfyk9umcemEoUHHEhFU7nIS1tQ38S+L3mXR+gYGFeZy39UTufWiSgpyde66SLpQuUtSItEYi9bv4ie/28rizXsp6ZfLvVdN4JaLKinO18dIJN3op1JOqHZvM8+9Xc+TS2rY3tTKyIH9uO/qidx0wSgNHyCSxlTu8gd2H2rjl+/s4Lm3t7N82z4AZp4+mG/MnszlZw4jrCtMRdKeyl1wd9ZuP8ArGxp4ZUMDq+r24w5nDCvm3qsmMPucEVSUFgYdU0Q+AJV7HxSJxtiw8yBLt+yNf23dy97D7ZjBlPKBfOXyM7hy8jDOPG1A0FFF5CSp3LNcS3uU6oZDbNh5gDX1TbxT38S6HQdo7YgBUFHaj8smDOXCcYO5dMIQyorzA04sIqmQVLmb2Szg+0AYeMTdv9lleT7wGHA+sAe4wd23pjaqHM/htgg7mlqo2dvMtj3NRx/fazhI3b4W3OPrFeaFOWtECTdNH805FSVMqyxlhC42EslK3Za7mYWBucBHgTpgmZktcPd1nVb7ArDP3U83sznAt4AbeiJwXxCLOYfaIzQ1d7CvuZ19zR3sO9zOnsPtNB5sY/ehNhoPtrHrQCvb97dwoDXyvu8vzAszqrSQcysG8enzKxg/tJjxw/ozpqxIB0NF+ohkttynA9XuvhnAzOYB1wKdy/1a4BuJ6WeAfzczcz+yzZhZ3J1ozInEnFhi+sjzSNTpiMYS0zE6Es87ojHaozHaI4mvxHRbJEZrR5TWjvhjS0eUlvYoze1RWjoiHGqLcrgtwuG2CAdbIxxs7eBgW4Tj/cvlho2y4nzKivMpH1TI9DGlnFbSj9NKCqgoLWT04EIGF+VpTBeRPi6Zch8J1HZ6XgdccLx13D1iZk3AYGB3KkJ2Nn9ZLQ+9vgkHSBSgAzF33MFxYrHEfPejy2JOfLk7UXdisfj6UY8XeCwWXy+aeJ2ekp8TojAvTL/cMP3ywhTn51BckMPgokKK83MY0C83/lUQnx5UmMegwlwGFuZRVpxHSb9cFbeIdKtXD6ia2R3AHQCjRo06qdcYVJTHxOEDwMDirwlAqNPzo4+JeeFQYjqxLBwyQnbk68hyIxzi6PyckBEKGeFQfPr3jyFywkZu2MgJhcgNh8jL6TwdIj8n/pgXDpGfG6IgJ0xBbpj8nJDuQiQivSKZcq8HOt9Cpzwx71jr1JlZDlBC/MDq+7j7w8DDAFVVVSe1ffzRScP46KRhJ/OtIiJ9RiiJdZYB481sjJnlAXOABV3WWQDckpi+DnglU/e3i4hkg2633BP70O8CXiR+KuSj7r7WzB4Elrv7AuBHwE/NrBrYS/wXgIiIBCSpfe7uvhBY2GXeA52mW4FPpzaaiIicrGR2y4iISIZRuYuIZCGVu4hIFlK5i4hkIZW7iEgWsqBORzezRmDbSX57GT0wtEGKpGu2dM0F6ZstXXNB+mZL11yQPdlGu/uQ7lYKrNxPhZktd/eqoHMcS7pmS9dckL7Z0jUXpG+2dM0FfS+bdsuIiGQhlbuISBbK1HJ/OOgAJ5Cu2dI1F6RvtnTNBembLV1zQR/LlpH73EVE5MQydctdREROIGPL3czONbPFZva2mS03s+lBZ+rMzL5sZhvMbK2ZfTvoPJ2Z2T1m5mZWFnSWI8zsO4l/r9Vm9gszGxhwnllmttHMqs3sviCzHGFmFWb2qpmtS3yu7g46U1dmFjazt8zs+aCzdGZmA83smcRnbL2ZXRh0JgAz+/PE/8s1ZvaUmRWk6rUzttyBbwN/6+7nAg8knqcFM7uM+H1lz3H3ycA/BRzpKDOrAK4EaoLO0sXLwFnuPgV4F7g/qCCdbgp/NTAJuNHMJgWVp5MIcI+7TwJmAF9Kk1yd3Q2sDzrEMXwf+F93nwicQxpkNLORwJ8BVe5+FvEh1VM2XHoml7sDAxLTJcD2ALN09UXgm+7eBuDuDQHn6ex7wF9y9A606cHdX3L3SOLpYuJ3/ArK0ZvCu3s7cOSm8IFy9x3uvjIxfZB4QY0MNtXvmVk58DHgkaCzdGZmJcDFxO87gbu3u/v+YFMdlQP0S9zBrpAU9lgml/tXgO+YWS3xLePAtvSO4Qzgw2a2xMx+bWbTgg4EYGbXAvXuviroLN34PPDLAN//WDeFT5sSBTCzSmAqsCTYJO/zL8Q3HGJBB+liDNAI/Dixy+gRMysKOpS71xPvrhpgB9Dk7i+l6vV79QbZH5SZLQKGH2PR14DLgT9395+b2fXEfytfkSbZcoBS4n86TwPmm9nY3rj1YDe5/or4LplAnCibuz+XWOdrxHc/PNGb2TKJmRUDPwe+4u4Hgs4DYGYfBxrcfYWZXRp0ni5ygPOAL7v7EjP7PnAf8DdBhjKzQcT/IhwD7AeeNrPPuvvjqXj9tC53dz9uWZvZY8T37wE8TS//KdhNti8CzybKfKmZxYiPHdEYVC4zO5v4h2iVmUF8t8dKM5vu7jt7OteJsh1hZrcCHwcuD/gevMncFD4QZpZLvNifcPdng87TyUxgtpldAxQAA8zscXf/bMC5IP6XV527H/kr5xni5R60K4At7t4IYGbPAhcBKSn3TN4tsx24JDH9EeC9ALN09d/AZQBmdgaQR8ADFrn7O+4+1N0r3b2S+Af+vN4q9u6Y2Szif9LPdvfmgOMkc1P4Xmfx38o/Ata7+3eDztOZu9/v7uWJz9Yc4JU0KXYSn/FaM5uQmHU5sC7ASEfUADPMrDDx//ZyUnigN6233LtxO/D9xIGIVuCOgPN09ijwqJmtAdqBWwLeEs0E/w7kAy8n/rJY7O53BhHkeDeFDyJLFzOBm4F3zOztxLy/StzjWE7sy8ATiV/Wm4HbAs5DYhfRM8BK4rsi3yKFV6rqClURkSyUybtlRETkOFTuIiJZSOUuIpKFVO4iIllI5S4ikoVU7iIiWUjlLiKShVTuIiJZ6P8DtkRmteUBWjsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sigmoid(x):\n",
    "    # 直接返回sigmoid函数\n",
    "    return 1. / (1. + np.exp(-x))\n",
    " \n",
    "# param:起点，终点，间距\n",
    "x = np.arange(-8, 8, 0.2)\n",
    "y = sigmoid(x)\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "针对手写数字识别的任务，网络层的设计如下：\n",
    "\n",
    "* 输入层的尺度为28×28，但批次计算的时候会统一加1个维度（大小为batch size）。\n",
    "* 中间的两个隐含层为10×10的结构，激活函数使用常见的Sigmoid函数。\n",
    "* 与房价预测模型一样，模型的输出是回归一个数字，输出层的尺寸设置成1。\n",
    "\n",
    "下述代码为经典全连接神经网络的实现。完成网络结构定义后，即可训练神经网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.nn.functional as F\n",
    "from paddle.nn import Linear\n",
    "\n",
    "# 定义多层全连接神经网络\n",
    "class MNIST(paddle.nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(MNIST, self).__init__()\n",
    "        # 定义两层全连接隐含层，输出维度是10，当前设定隐含节点数为10，可根据任务调整\n",
    "        self.fc1 = Linear(in_features=784, out_features=10)\n",
    "        self.fc2 = Linear(in_features=10, out_features=10)\n",
    "        # 定义一层全连接输出层，输出维度是1\n",
    "        self.fc3 = Linear(in_features=10, out_features=1)\n",
    "    \n",
    "    # 定义网络的前向计算，隐含层激活函数为sigmoid，输出层不使用激活函数\n",
    "    def forward(self, inputs):\n",
    "        # inputs = paddle.reshape(inputs, [inputs.shape[0], 784])\n",
    "        outputs1 = self.fc1(inputs)\n",
    "        outputs1 = F.sigmoid(outputs1)\n",
    "        outputs2 = self.fc2(outputs1)\n",
    "        outputs2 = F.sigmoid(outputs2)\n",
    "        outputs_final = self.fc3(outputs2)\n",
    "        return outputs_final"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 卷积神经网络\n",
    "\n",
    "虽然使用经典的全连接神经网络可以提升一定的准确率，但其输入数据的形式导致丢失了图像像素间的空间信息，这影响了网络对图像内容的理解。对于计算机视觉问题，效果最好的模型仍然是卷积神经网络。卷积神经网络针对视觉问题的特点进行了网络结构优化，可以直接处理原始形式的图像数据，保留像素间的空间信息，因此更适合处理视觉问题。\n",
    "\n",
    "卷积神经网络由多个卷积层和池化层组成，如 **图4** 所示。卷积层负责对输入进行扫描以生成更抽象的特征表示，池化层对这些特征表示进行过滤，保留最关键的特征信息。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/91f3755dfd47461aa04567e73474a3ca56107402feb841a592ddaa7dfcbc67c2\" width=\"800\" hegiht=\"\" ></center>\n",
    "\n",
    "<center><br>图4：在处理计算机视觉任务中大放异彩的卷积神经网络</br></center>\n",
    "<br></br>\n",
    "\n",
    "\n",
    "-------\n",
    "**说明：**\n",
    "\n",
    "本节只简单介绍用卷积神经网络实现手写数字识别任务，以及它带来的效果提升。读者可以将卷积神经网络先简单的理解成是一种比经典的全连接神经网络更强大的模型即可，更详细的原理和实现在接下来的《计算机视觉-卷积神经网络基础》中讲述。\n",
    "\n",
    "------\n",
    "\n",
    "两层卷积和池化的神经网络实现如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 定义 SimpleNet 网络结构\n",
    "import paddle\n",
    "from paddle.nn import Conv2D, MaxPool2D, Linear\n",
    "import paddle.nn.functional as F\n",
    "# 多层卷积神经网络实现\n",
    "class MNIST(paddle.nn.Layer):\n",
    "     def __init__(self):\n",
    "         super(MNIST, self).__init__()\n",
    "         \n",
    "         # 定义卷积层，输出特征通道out_channels设置为20，卷积核的大小kernel_size为5，卷积步长stride=1，padding=2\n",
    "         self.conv1 = Conv2D(in_channels=1, out_channels=20, kernel_size=5, stride=1, padding=2)\n",
    "         # 定义池化层，池化核的大小kernel_size为2，池化步长为2\n",
    "         self.max_pool1 = MaxPool2D(kernel_size=2, stride=2)\n",
    "         # 定义卷积层，输出特征通道out_channels设置为20，卷积核的大小kernel_size为5，卷积步长stride=1，padding=2\n",
    "         self.conv2 = Conv2D(in_channels=20, out_channels=20, kernel_size=5, stride=1, padding=2)\n",
    "         # 定义池化层，池化核的大小kernel_size为2，池化步长为2\n",
    "         self.max_pool2 = MaxPool2D(kernel_size=2, stride=2)\n",
    "         # 定义一层全连接层，输出维度是1\n",
    "         self.fc = Linear(in_features=980, out_features=1)\n",
    "         \n",
    "    # 定义网络前向计算过程，卷积后紧接着使用池化层，最后使用全连接层计算最终输出\n",
    "    # 卷积层激活函数使用Relu，全连接层不使用激活函数\n",
    "     def forward(self, inputs):\n",
    "         x = self.conv1(inputs)\n",
    "         x = F.relu(x)\n",
    "         x = self.max_pool1(x)\n",
    "         x = self.conv2(x)\n",
    "         x = F.relu(x)\n",
    "         x = self.max_pool2(x)\n",
    "         x = paddle.reshape(x, [x.shape[0], -1])\n",
    "         x = self.fc(x)\n",
    "         return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "\n",
    "使用MNIST数据集训练定义好的卷积神经网络，如下所示。  \n",
    "\n",
    "------\n",
    "\n",
    "**说明：**  \n",
    "以上数据加载函数load_data返回一个数据迭代器train_loader，该train_loader在每次迭代时的数据shape为[batch_size, 784]，因此需要将该数据形式reshape为图像数据形式[batch_size, 1, 28, 28]，其中第二维代表图像的通道数（在MNIST数据集中每张图片的通道数为1，传统RGB图片通道数为3）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading mnist dataset from ./work/mnist.json.gz ......\n",
      "mnist dataset load done\n",
      "epoch: 0, batch: 0, loss is: 25.196237564086914\n",
      "epoch: 0, batch: 200, loss is: 2.8643529415130615\n",
      "epoch: 0, batch: 400, loss is: 2.0646779537200928\n",
      "epoch: 1, batch: 0, loss is: 3.135349988937378\n",
      "epoch: 1, batch: 200, loss is: 2.058072090148926\n",
      "epoch: 1, batch: 400, loss is: 2.080343723297119\n",
      "epoch: 2, batch: 0, loss is: 1.9587202072143555\n",
      "epoch: 2, batch: 200, loss is: 1.6729546785354614\n",
      "epoch: 2, batch: 400, loss is: 1.7185478210449219\n",
      "epoch: 3, batch: 0, loss is: 1.4882879257202148\n",
      "epoch: 3, batch: 200, loss is: 1.239805817604065\n",
      "epoch: 3, batch: 400, loss is: 1.5459805727005005\n",
      "epoch: 4, batch: 0, loss is: 2.2185895442962646\n",
      "epoch: 4, batch: 200, loss is: 1.598059058189392\n",
      "epoch: 4, batch: 400, loss is: 1.8100342750549316\n",
      "epoch: 5, batch: 0, loss is: 1.324904441833496\n",
      "epoch: 5, batch: 200, loss is: 1.1214401721954346\n",
      "epoch: 5, batch: 400, loss is: 1.9421234130859375\n",
      "epoch: 6, batch: 0, loss is: 1.0814441442489624\n",
      "epoch: 6, batch: 200, loss is: 1.5564398765563965\n",
      "epoch: 6, batch: 400, loss is: 0.9601972699165344\n",
      "epoch: 7, batch: 0, loss is: 1.287195086479187\n",
      "epoch: 7, batch: 200, loss is: 1.1438658237457275\n",
      "epoch: 7, batch: 400, loss is: 1.0299162864685059\n",
      "epoch: 8, batch: 0, loss is: 1.0495307445526123\n",
      "epoch: 8, batch: 200, loss is: 1.5844645500183105\n",
      "epoch: 8, batch: 400, loss is: 0.9159772992134094\n",
      "epoch: 9, batch: 0, loss is: 0.8777803778648376\n",
      "epoch: 9, batch: 200, loss is: 1.1280484199523926\n",
      "epoch: 9, batch: 400, loss is: 1.1104599237442017\n"
     ]
    }
   ],
   "source": [
    "#网络结构部分之后的代码，保持不变\n",
    "def train(model):\n",
    "    model.train()\n",
    "    #调用加载数据的函数，获得MNIST训练数据集\n",
    "    train_loader = load_data('train')\n",
    "    # 使用SGD优化器，learning_rate设置为0.01\n",
    "    opt = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters())\n",
    "    # 训练5轮\n",
    "    EPOCH_NUM = 10\n",
    "    # MNIST图像高和宽\n",
    "    IMG_ROWS, IMG_COLS = 28, 28\n",
    "    loss_list = []\n",
    "    for epoch_id in range(EPOCH_NUM):\n",
    "        for batch_id, data in enumerate(train_loader()):\n",
    "            #准备数据\n",
    "            images, labels = data\n",
    "            images = paddle.to_tensor(images)\n",
    "            labels = paddle.to_tensor(labels)\n",
    "            \n",
    "            #前向计算的过程\n",
    "            predicts = model(images)\n",
    "            \n",
    "            #计算损失，取一个批次样本损失的平均值\n",
    "            loss = F.square_error_cost(predicts, labels)\n",
    "            avg_loss = paddle.mean(loss)\n",
    "\n",
    "            #每训练200批次的数据，打印下当前Loss的情况\n",
    "            if batch_id % 200 == 0:\n",
    "                loss = avg_loss.numpy()[0]\n",
    "                loss_list.append(loss)\n",
    "                print(\"epoch: {}, batch: {}, loss is: {}\".format(epoch_id, batch_id, loss))\n",
    "            \n",
    "            #后向传播，更新参数的过程\n",
    "            avg_loss.backward()\n",
    "            # 最小化loss,更新参数\n",
    "            opt.step()\n",
    "            # 清除梯度\n",
    "            opt.clear_grad()\n",
    "\n",
    "    #保存模型参数\n",
    "    paddle.save(model.state_dict(), 'mnist.pdparams')\n",
    "    return loss_list\n",
    "\n",
    "model = MNIST()\n",
    "loss_list = train(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "可视化损失变化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAFDCAYAAACUUBKvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt4XHW97/HPdy5JZtJ7kl4olJa2FEopBUPl5gYUBNwoF7du8KDgphb2lr0RPW7YHJ8j7nNE8REB8YrAhn3kIgqCinIXEFCwlAotLfQCLZS2Se9pc5vL9/wx0yQtSZrJrJmVpO/X8+SZNWtWZr5Zmaf59Pdb8/uauwsAAADhiYRdAAAAwL6OQAYAABAyAhkAAEDICGQAAAAhI5ABAACEjEAGAAAQMgIZAABAyAhkAAAAISOQAQAAhCwWdgGFqq2t9cmTJ4ddBgAAwF69/PLLG929bm/HDbpANnnyZC1YsCDsMgAAAPbKzFb35TimLAEAAEJGIAMAAAgZgQwAACBkBDIAAICQEcgAAABCNug+ZQkAwGCyfft2NTQ0KJVKhV0KAhaPxzV27FiNGDGi6OcikAEAUCLbt2/Xhg0bNHHiRCUSCZlZ2CUhIO6ulpYWrV27VpKKDmVMWQIAUCINDQ2aOHGikskkYWyIMTMlk0lNnDhRDQ0NRT8fgQwAgBJJpVJKJBJhl4ESSiQSgUxHlyWQmdkBZvZHM3vdzJaY2eX5/deY2VozW5T/+lg56ulNavEG7fjRS/JMNuxSAABDACNjQ1tQv99yjZClJX3F3WdKOkbSF81sZv6xG9x9Tv7r92Wqp0dtT72lbV/8vbJbW8MuBQAA7CPKEsjcfZ27L8xvN0laKmliOV67UJHapCQpu7E55EoAAMC+ouzXkJnZZElHSnoxv+syM3vVzG43s9E9fM98M1tgZgsaGxtLWl+khkAGAECpLVu2TGamBQsWFPU8jzzyiMxMGzduDKiycJQ1kJnZMEn3S/qSu2+X9GNJUyXNkbRO0vXdfZ+73+Lu9e5eX1dXV9IaGSEDACB3bVRvX5MnTy7q+adPn65169Zpzpw5wRQ8yJVtHTIziysXxu5y9wckyd03dHn8Z5J+V656etIRyDYRyAAA+65169Z1bL/wwgv65Cc/qYULF2rChAmSpGg02u33tbe3q6KiYq/PH41GNX78+GCKHQLK9SlLk3SbpKXu/r0u+yd0OewcSYvLUU9vGCEDAEAaP358x9eYMWMkSXV1dR37ds1YjR8/Xt/4xjc0f/58jRkzRqeeeqok6bvf/a5mz56t6upq7bfffrrgggt2W69rzynLXfcfeOABnXHGGUomk5o2bZruvvvugmt/7rnndMIJJ6iqqkpjxozR5z73OW3atKnj8dWrV+vss89WTU2NEomEpk2bpptuuqnj8V/96lc64ogjlEwmNXr0aB177LFavLi0EaVcI2THS/qspNfMbFF+39WSzjezOZJc0tuSLilTPT2yZFyqjBLIAAAlsfVLf1Bq0fqyv258zniNuvGMkjz39ddfr6uuukovvviiMpmMpNyU54033qgpU6bovffe0xVXXKHPfvazevTRR3t9riuvvFLXXXedbr75Zv3kJz/RRRddpOOOO67PU6TvvPOOTjvtNH3qU5/ST3/6U23cuFGXXnqpzjvvPD3++OOSpC984QuKRqN66qmnNHLkSK1cubIjsK1Zs0bnnXeerr/+en3iE59QS0uLXn755R5HBINSlkDm7s9J6m6hjtCXudiTmSlSmySQAQDQRx/60Id09dVX77bvK1/5Ssf2lClTdNNNN+m4447Tpk2bVFNT0+NzXXHFFTr33HMlSddee61uvvlmPfPMM30OZN///vc1btw43XrrrYrFcjHnjjvu0DHHHKOXXnpJc+fO1erVq/X5z39eRxxxhCTt9txr165VNpvVpz/96Y7p2ZkzZ77vdYJGL8tuRGuTym5qCbsMAMAQVKpRqjDNnTv3ffueeOIJXXfddVq2bJm2bt2qbDa34Prq1at7DWRdL/KvqKhQbW2tNmzY0OPxe1qyZImOO+64jjC2q76qqiotWbJEc+fO1Ze//GVddtlleuihh3TSSSfpzDPP1PHHHy9JOvroo3XiiSdqxowZOvXUU3XSSSfp3HPP1cSJpV2ti9ZJ3YjUMEIGAEBfVVdX73Z/xYoVOvPMMzVjxgz94he/0IIFC/TLX/5SUu6i/97s+YEAM+sIc0G55JJL9NZbb+niiy/WmjVrdOqpp2revHmSpFgspqeeekqPPfaYjjzySN17772aPn16x3RnqRDIusGUJQAA/ffiiy8qlUrpxhtv1HHHHacZM2Zo/fryXDd32GGH6YUXXlA6ne7Y99JLL6m1tVWzZs3q2Lf//vtr3rx5uuuuu/SjH/1It99+u9ra2iTlQuAxxxyjr33ta3r++ec1d+5c3XHHHSWtm0DWjUhtkmUvAADop4MPPljZbFY33HCD3nrrLd1///361re+VZbXvvzyy7VhwwbNmzdPS5Ys0TPPPKPPf/7zOuWUU3T00UdLki699FI98sgjWrlypRYvXqwHH3xQU6dOVWVlpZ5++mlde+21eumll7RmzRo99thjev3110t+HRmBrBuR2qSym1toMA4AQD8cffTR+t73vqebbrpJM2fO1M0336wbbrihLK+9//7769FHH9Xy5cv1gQ98QOecc47q6+t17733dhyTyWT0r//6r5o1a5ZOPPFEZTIZ/fa3v5UkjR49Ws8++6w+/vGPa/r06Zo/f74uvvhiXXnllSWt29y9pC8QtPr6ei+2zcLe7Pj+X7Tt8kc0vvGritZW7/0bAADoxtKlS3XooYeGXQZKrLffs5m97O71e3sORsi6weKwAACgnAhk3ehsn8TSFwAAoPQIZN1ghAwAAJQTgawbkRoCGQAAKB8CWTcYIQMABGWwfXgOhQnq90sg64Yl41JVjLXIAABFicfjamnheuShrKWlRfF4vOjnIZB1w8xy/SwZIQMAFGHs2LFau3atmpubGSkbYtxdzc3NWrt2rcaOHVv089FcvAeRmgSBDABQlBEjRkiS3nvvPaVSqZCrQdDi8bjGjRvX8XsuBoGsB/SzBAAEYcSIEYH8wcbQxpRlD3L9LJn3BwAApUcg60GkNqkMI2QAAKAMCGQ9iNQk5VtoMA4AAEqPQNaDSG1Scim7hWlLAABQWgSyHrA4LAAAKBcCWQ8iNQlJBDIAAFB6BLIeMEIGAADKhUDWg45AxtIXAACgxAhkPWCEDAAAlAuBrAeRZEWuwTiBDAAAlBiBrBc0GAcAAOVAIOtFrn0SgQwAAJQWgawXNBgHAADlQCDrRaQmQT9LAABQcgSyXjBCBgAAyoFA1otIbVK+tVWezoRdCgAAGMIIZL3obDDeGnYpAABgCCOQ9SJSw+KwAACg9Ahkvehsn0QgAwAApUMg6wXtkwAAQDkQyHpBIAMAAOVAIOtFpCYhiUAGAABKi0DWi0iyQpaIKbupJexSAADAEEYg2wsWhwUAAKVGINuLSA2BDAAAlBaBbC8itUn6WQIAgJIikO1FpDbJOmQAAKCkyhLIzOwAM/ujmb1uZkvM7PL8/jFm9riZLc/fji5HPYXgGjIAAFBq5RohS0v6irvPlHSMpC+a2UxJV0l60t2nS3oyf39AidQk5FtoMA4AAEqnLIHM3de5+8L8dpOkpZImSjpL0p35w+6UdHY56ilEx+Kwm1n6AgAAlEbZryEzs8mSjpT0oqRx7r4u/9B6SeN6+J75ZrbAzBY0NjaWpc5dOvtZEsgAAEBplDWQmdkwSfdL+pK7b+/6mLu7JO/u+9z9Fnevd/f6urq6MlTaifZJAACg1MoWyMwsrlwYu8vdH8jv3mBmE/KPT5DUUK56+ipSQyADAAClVa5PWZqk2yQtdffvdXnoN5IuzG9fKOmhctRTCEbIAABAqcXK9DrHS/qspNfMbFF+39WSvi3pPjO7WNJqSZ8uUz191tFgnLXIAABAiZQlkLn7c5Ksh4c/Uo4a+iuSrJAl44yQAQCAkmGl/j6I1CQIZAAAoGQIZH2Qa5/EshcAAKA0CGR9QINxAABQSgSyPqCfJQAAKCUCWR9EaghkAACgdAhkfRCpTcq30mAcAACUBoGsD2gwDgAASolA1gcdi8MybQkAAEqAQNYHtE8CAAClRCDrg45AxlpkAACgBAhkfRBlhAwAAJQQgawPIjUEMgAAUDoEsj6wRJwG4wAAoGQIZH2U62dJIAMAAMEjkPUR/SwBAECpEMj6KFKTYMoSAACUBIGsj3JTlix7AQAAgkcg66NILQ3GAQBAaRDI+qijwXiKBuMAACBYBLI+6liLjAbjAAAgYASyPupsn8S0JQAACBaBrI9oMA4AAEqFQNZH9LMEAAClQiDro0hNQhKBDAAABI9A1kcdF/WzFhkAAAgYgayPLBGXVdNgHAAABI9AVoBIDf0sAQBA8AhkBWC1fgAAUAoEsgLk+lkSyAAAQLAIZAVghAwAAJQCgawAkZoEgQwAAASOQFaASG1Svq2NBuMAACBQBLICdLRPosE4AAAIEIGsAPSzBAAApUAgK0DHav0EMgAAECACWQE6Goyz9AUAAAgQgawATFkCAIBSIJAVIFKTkEQgAwAAwSKQFcCqcg3G6WcJAACCRCArUK59EsteAACA4BDICkT7JAAAEDQCWYEiNQQyAAAQrLIEMjO73cwazGxxl33XmNlaM1uU//pYOWopFiNkAAAgaOUaIbtD0und7L/B3efkv35fplqKkruGjEAGAACCU5ZA5u7PStpcjtcqNRqMAwCAoIV9DdllZvZqfkpzdE8Hmdl8M1tgZgsaGxvLWd/7dKxFxigZAAAISJiB7MeSpkqaI2mdpOt7OtDdb3H3enevr6urK1d93WK1fgAAELTQApm7b3D3jLtnJf1M0tywailEZz9L1iIDAADBCC2QmdmELnfPkbS4p2MHEkbIAABA0GLleBEzu0fSSZJqzexdSV+XdJKZzZHkkt6WdEk5ailWpIZABgAAglWWQObu53ez+7ZyvHbQuKgfAAAELexPWQ46VhWXDaugwTgAAAgMgawfWK0fAAAEiUDWD5GaBIEMAAAEhkDWD7n2SSx7AQAAgkEg6wemLAEAQJAIZP1AIAMAAEHqcyAzs5PNbEp+e4KZ3Wlm/2Vm40tX3sAUqUnKt7fJ29NhlwIAAIaAQkbIfiQpk9++XlJcUlbSLUEXNdB1rNa/mevIAABA8QpZGHaiu68xs5ik0yQdKKld0nslqWwAi3ZpnxQdPzzkagAAwGBXSCDbbmbjJM2S9Lq77zCzCuVGyvYpHav1cx0ZAAAIQCGB7GZJf5VUIelL+X3HS1oWdFEDHQ3GAQBAkPocyNz9OjP7taSMu6/M714raV5JKhvAOgIZa5EBAIAAFNRc3N3f3LVtZidLyrr7M4FXNcBFahghAwAAwSlk2YtnzOz4/PaVku6VdLeZXV2q4gYqq4zRYBwAAASmkGUvZkn6S377C5JOlnSMpEuDLmowyLVPIpABAIDiFTJlGZHkZjZVkrn765JkZqNLUtkAx2r9AAAgKIUEsuck/UDSBEm/lqR8ONtYgroGPAIZAAAISiFTlhdJ2irpVUnX5PcdIummYEsaHCI1CQIZAAAIRCHLXmySdPUe+x4OvKJBIncNGcteAACA4hXyKcu4mX3DzFaZWWv+9hv51fr3OZFaGowDAIBgFDJl+R1Jpyj3qcoj8rcflnRdCeoa8KIsDgsAAAJSyEX9n5J0RH7qUpLeMLOFkv4m6YrAKxvgui4OG51Ag3EAANB/hYyQWYH7h7TO9klc2A8AAIpTSCD7paTfmtlpZnaomZ0u6cH8/n0ODcYBAEBQCpmy/HdJX5P0Q0n7KddY/F5J/6cEdQ14BDIAABCUXgOZmX14j11P579Mkuf3nSDpqaALG+giNQlJop8lAAAo2t5GyG7rYf+uMLYrmB0UWEWDhFXEZMMr+JQlAAAoWq+BzN2nlKuQwYj2SQAAIAiFXNSPPURqCGQAAKB4BLIi5NonEcgAAEBxCGRFYMoSAAAEgUBWBAIZAAAIAoGsCJGahLypnQbjAACgKASyItBgHAAABIFAVgRW6wcAAEEgkBWBQAYAAIJAICtCpIZABgAAikcgK0LHCBlrkQEAgCIQyIpAg3EAABAEAlkRrCImG1HJlCUAACgKgaxIkZoEgQwAABSlLIHMzG43swYzW9xl3xgze9zMludvR5ejlqDl+lmyDhkAAOi/co2Q3SHp9D32XSXpSXefLunJ/P1Bh/ZJAACgWGUJZO7+rKTNe+w+S9Kd+e07JZ1djlqCRiADAADFCvMasnHuvi6/vV7SuBBr6bdITZJlLwAAQFEGxEX97u6SvKfHzWy+mS0wswWNjY1lrGzvorXJXIPxNhqMAwCA/gkzkG0wswmSlL9t6OlAd7/F3evdvb6urq5sBfYFi8MCAIBihRnIfiPpwvz2hZIeCrGWftu1OCzXkQEAgP4q17IX90j6s6QZZvaumV0s6duSTjWz5ZJOyd8fdDpHyFj6AgAA9E+sHC/i7uf38NBHyvH6pdQRyBghAwAA/TQgLuofzHYFMvpZAgCA/iKQFSkyhmvIAABAcQhkRepoMM6nLAEAQD8RyALAav0AAKAYBLIAEMgAAEAxCGQBiNQkCGQAAKDfCGQBiNQmWYcMAAD0G4EsAExZAgCAYhDIAhCtTcp3tMtbU2GXAgAABiECWQAiNbRPAgAA/UcgC0BnP0umLQEAQOEIZAGgnyUAACgGgSwAkZpc+yT6WQIAgP4gkAWgc8qSa8gAAEDhCGQB6LionxEyAADQDwSyAFg8KhtZSSADAAD9QiALSKSGxWEBAED/EMgCkmufRCADAACFI5AFhPZJAACgvwhkASGQAQCA/iKQBSRSkyCQAQCAfiGQBSRam5TvTNFgHAAAFIxAFhAWhwUAAP1FIAsI/SwBAEB/EcgCsmu1fvpZAgCAQhHIAtI5ZUkgAwAAhSGQBYQpSwAA0F8EsoBExiQkEcgAAEDhCGQB6WgwzqcsAQBAgQhkAWK1fgAA0B8EsgARyAAAQH8QyAIUqSGQAQCAwhHIAhSpTbLsBQAAKBiBLEBRpiwBAEA/EMgCFNnVYLyFBuMAAKDvCGQBitTk1yJj2hIAABSAQBagzvZJrEUGAAD6jkAWoF2BjAbjAACgEASyANHPEgAA9AeBLECRGgIZAAAoHIEsQB0NxrmoHwAAFIBAFiCLR2WjqhghAwAABYmFXYCZvS2pSVJGUtrd68OtqDj0swQAAIUKPZDlnezuG8MuIgiRmgTLXgAAgIIwZRkwRsgAAEChBkIgc0mPmdnLZjY/7GKKRT9LAABQqIEwZXmCu681s7GSHjezZe7+bNcD8kFtviRNmjQpjBr7jBEyAABQqNBHyNx9bf62QdKvJc3t5phb3L3e3evr6urKXWJBIjVJeTMNxgEAQN+FGsjMrNrMhu/alvRRSYvDrKlYnf0sGSUDAAB9E/aU5ThJvzazXbXc7e6PhFtScbr2s4zuPzLkagAAwGAQaiBz91WSjgizhqBFavKr9XMdGQAA6KPQryEbajqnLFmLDAAA9A2BLGAdgYwRMgAA0EcEsoB1NBgnkAEAgD4ikAXMYjQYBwAAhSGQlUCkNsmyFwAAoM8IZCXAav0AAKAQBLISoJ8lAAAoBIGsBCI1CQIZAADoMwJZCeSuIWMdMgAA0DcEshKI1OYajGeb28MuBQAADAIEshJgtX4AAFAIAlkJRGp2BTKuIwMAAHtHICuBgdY+Kb1qs7I7mT4FAGCgIpCVwEAJZNltrdp62cPaMO37ajjkB2p5aFmo9QAAgO4RyEog7EDm7mr55RJtOPQH2vmjvyr5hQ/IRldp89n3atM59yrz7rZQ6gIAAN2LhV3AUBQZXSVJSq/cIneXmZXttdNvbdHWLz6stj+sUPyoCar5zfmqqJ8oT2W044Y/q+map7Xh0B9qxDc/rOovzpVFyeQAAISNv8YlYLGo4nMnaueNf1Hjsbeq5ddL5dlsSV/TUxk1fftPajjsh2r/0xqNvPF01b04TxX1E3M1xaMa/u8naOySL6rihEnadvkjavzgz9S+8L2S1gUAAPbO3D3sGgpSX1/vCxYsCLuMvfKWlHbesUg7vvuCMqu2KDajRsO+erySF8yWVQY7MNn2/BptveS3Si9pVNU5h2jU989QdP+RPdfmrpb7lmjb5X9QtrFZ1Zd/UCP+82RFhlUGWhcAAPs6M3vZ3ev3ehyBrLQ8nVHL/Uu147rnlHplvSIThmnYl45R9aX1ioyoKuq5s5ubte2qJ9T8s4WKThqpkT/4mBIfn9H379/aou3/8aR2/mSBogeMyH3/Jw4pqiYAANCJQDbAuLvanlilHdc9p7Yn35KNqFT1P9dr2OXHKDpheMHP1XLXq9r25UeV3dyiYVccq+FfP7HfI1xtL6zR1kt+p/TiBlWde2huhG3iiH49FwAA6EQgG8DaF6zVju88r5b7l0qxiJIXHqHhXz1esek1e/3e1Jsbte1fHlbbk28p/sGJGv3Tjyt+xPiia/JURjuuf0Hbv/GMLB7RiG9+RNX/cjQX/QMAUAQC2SCQXrFJTd99Qc13LJLaM6o691ANv/IEVRw98X3HeltaTdc9p6Zr/ySrimnEt05R9fwPBB6Y0qs2a+s/P6y2x1YqfvR+GnXLx1UxZ0KgrwEAwL6CQDaIZNY3acfNL2nnj/4q39qqipMna/iVJ6jyo1NlZmp7+i1tvfR3Sr+xSYl/PEwjbzi94GnOQri7Wn6xWNsuf0TZTc0a9qVjNPyak7joH3vlqYxaH35TO29dqMyqLUqce6gSF8xW/JC6sEsDgFAQyAahbFObdt7ysnbc8Gdl1zYpfsQ4xWbUquW+JYpOGaVRPz5TVadNK189W1pyHxq45WVFJ43MhbJRVVImK09npYz3uO2Z/L50dvftrCtSHVdkTEI2JqHImIQiNcnc7ZiEbFhFWddtQzDSyzdp520L1XzHImU37FRkv+GKH1KrtqfflrKueP1+Sl4wW4nzZik6bljY5QJA2RDIBjFvT6v5rte04zvPK71is4Z99TgN/9rfKZKsCKWetufXaOv83yr9emNxT2SSIpYLZz2JRTrCWUdI63q/JncbHVut2KF1iowfRoALibek1HL/69p560K1P7NaipqqPj5D1fOOUuVpU2WxqDLrm9Ry72I1//xVpV5eJ0VNlR+dquQFs1V11iGKVIfznh7svDUlVcZ47wODAIFsCPBsVmrPyKriYZciT2WUWtIgSbnr1mIRKWod2xY1qbftqMkiuevdvDWl7JZWZTe37P61qblj2/d8bHOLvOn9DdJtdJXiM+sUO2xs/rZO8Zl1ikwYzh+rEmlftE7Nty5U889flW9rU3TqaFXPO0rJC+f0OpWeer1BzXe9ppafv6rMmm2y6riqzj1UyQtmq/IjB/EBkj5ILd6gpm8/p5Z7Fit+5HgNv+YkVf39wbzXgQGMQIYhx1MZZbfkwllmbZPSrzcq/XqjUksalFrSKN/c0nGsjaraLaDtCmyR/Qhq/ZHd1qqWe17TzlsX5ka6KqNK/MNMVc87ShUnTi7onHo2q/bn1qj556+q5b4l8m1tikwYpuT5h+euN5szvl+/I3dXdmOzMm9vVWb1VqXf3qrM6m3KrN4qz7gqPzJFVWdMV+yQ2kH3Hmh/6V01XfsntT70hqw6rsRnDlfbk28ps2qL4h+YQDADBjACGfYp7q5sw86OgJa7bVR6SYOym7oEtZGVHQEtdmitLBGX2jPytrS8LSO1peXtGXlbbp/yt96e6dxuy+Tv578nFlFsyihFDxqtWP4rOnWMYpNHBd6VoZzcXe3Pr1HzrQtzwaklrdjscar+wlFK/o/ZioxOFP8arSm1PrxczT9/Va0PvymlsorNrFPys7OV+Mzhik0a1XlsNqvshp35oLU1H7y2dd5fvU3enNrt+W1kpWKTR8nbMkov2yhJik4aqcrTp6nq9Gmq/MiUohdoLhV3V/sf31LTtX/KrV04ukrDLj9G1ZfNVbQmKU9l1PzzV9X0f58lmAEDGIEMyMs07OgMaLsC25JGZTc2v/9gU+7anMqoLH+rii7bXR+riEqVUak9o/RbW5VZtWX3QGBSdOKIXFCbOvp9gS1SmxyQfzgz65vU/PNX1XzrQqXf2CQbXqHEZw5X9byjFP/AfiWrObu5WS2/fF3N/+9van/+HUlSxQmTZFUxpfOBS+2Z3b4nUpNQdPIoRQ8cpdjkUYoeOFLRyaMUOzC3HRnVGRrTa7aq7ZEVan1khdqeWJWbAo9FVHHcAblwdsY0xY/o3+hckDybVevv3lTTtX9S6sW1iowfpmFfOVbVl9QrMvz9n3QOM5i5uzIrNyt6wMhB/Z+PocrdlV7coLZnV6vimP0VP2pC6O/vfRGBDNiLzMadUirbGbQqornr3vr5D1bHKN3Kzcqs2qL0qi2525W57ex7Tbsdb8MqOkPa1Pyo2vQxik2vUfSAER3X3JWKt6eVXrZRqVc3KPVaQ/52g7Jrc3VWHH+AkvOOUuJTh5X94vv0qs1qvvs1td6/VKqIvj9s5e/3dykWT2XU/ud3cuHsDyuUWrRekhQZP0xVp01V5enTVHnqVEVrkkH+WL3XlM6o5b4lavrWc0ovblB08igNv/J4JS+a06frSMsVzDINO9T22Eq1PrZSbY+tVHbDTtmoqtwSJ+fPUuXJU7geMESezqj9hXfU8uAytT70hjKrtnQ8Fj9inJLzghvhRt8QyIABxltSSr+9tcfAptZ058GVUcW6BLSut5H9hhcU1txdmXe3K/3aht3CV3rZRimdzR0Uj+Smcg8fp/jhY1V15sGKzxwb8BkYuDLrmnIB45EVan1sZe56RJPicyeq6ozpqjp9muL1+5UkaHhbWs13LlLTdc8rs2qLYjPrNPw/TlDivFmyWLTw5ws4mHl7Wu3Pv6PWR1eo7bGVSr2SD6+1SVWeepAqTpik9r+8q9ZfL5PvaFdkXLUSnzpMifNnqeLYAxiRKYNsc7vaHl+l1geXqfW3b+Qu06iIqvKUg5Q4+xBVnjRZrU/W5EhmAAAMoUlEQVSsUvOtC5VamL8G9JMzVX3xkao4aXLJ//O3ryOQAYOIZ7PKvtek9PLNSi/ftPvtys1SW+dUnSViik7rEtS6bNuwitzU7K7wlQ9gvrW14/ujk0YqdvhYxWePU/zwcYrPHqfYwTWyeOF//Iciz2SVWvCeWv+wXK2PrFDqpbWSSzYmofjMOkUnjVT0wJGKTRqZ3x6l6KSR3U4n9ia7I7/u4PV/Vva9JsXr99Pw//UhVX1iRiB/IPsbzNxd6eWb1PboSrU+ukLtT78t35nqnN49baoqT5um+JHjd6vTW1JqffhNNd+7WK2/e1Nqyyh64Egl/nGWEufPKul0cGZTs1KvrFP69UZZVUyRumpFxlYrUpdUtK5aNqpqyAXDTONOtf7uTbU+tExtj62Ut6Rlo6pU9ffTcyHstGndvifbX1mn5tsWqvmu1+RbWxU9aLSq/+lIJS+aQw/jEiGQAUOEZ7O5Ea7lm5XZM6yt2vK+66p2seEVnYFrVwCbNXa366qwd5lNzWp7fKXanliV+x2s2abMu9s7RxfzbFRVLuwemA9qXQJbbNLI3Jp50YiyW1q04wcvaedNf1F2U0uuM8fVH8ot/VGC0NCXYJbd2qK2p95S66Mr1fboitz1epKi08bkAthHp6ry5Cl9Dp3Z7a1qfegNNd/zmtoeWyllXLFDapU4Lx/ODq7t38/irszqrUq9sl6pReuVemWdUq+sz/0+ehOLKFKXVKSuWtH8bdfAtmt71+M2OjEgA1x65Wa1PLRMrQ8uy11nmXVFDxihqrMPUdVZh6jy7w7s83+svCWllgeWaudtC9X+x7eliKny9GmqnneUqs48mP+gBYhABuwDPJNVZs22joDmTW25JT4OH6vogaMG5B+VocAzWWXX78h92GDNts6v1duUzm93HZWUJMUiiu4/QtmNzfId7ao682ANu/pDqjz2gPLU3E0wqzxtmtqfflvtL74rZVw2vEKVHzmoI4TFDhpT9OtmNu5U6/1L1XzPa2p/drXkUvzI8Uqcf7gS/3jYbp+k3bPe9LKNSr2yTu2L1neEsI7zGjHFZtQoPme84kdOUPzI8YrPGiuls8o0NivbsFPZxp3KNjYr07gzf79Z2cadyuS3fXtb90VHTDaqSpHRVYqMTnTc2h73I6OrZHveH1HZ7xFOd5eyua4nyrg860ovbVTrQ2+o5cFlSi/OrQUZmz1OibMPUdXZh/R7mZiu0is3a+ftr+Q6bbzXpMjYaiU/d4SSFx8Zetuz7I62jv+EZpZvVurNTcos35Sblo1abrHxaH7Ny262c2tg5rcj1rl+Zn572JePVeVxk0r6MxDIACBE2e2tu4W1XUHNKmMa9m8fVPyI8aHUtVswe2uL4kdP7AhgFR/cv6QjI5m129Vy3xI13/OaUn99T1LuwyOJ8w9XfPY4pV7bkAter6xTanFD51R9VSw3wjtnvCqOzAWw2OFji+5e4m3p3Np1XcJbtnGnshubc4tXb2npuPUu9/ccHd2NSTayKtdmLmKd4SrTpcVc/lZZ79zO5MNYdyKmir87UImzZqjqrEMUmzK6qJ+7x/ORzqj1kRVqvu2V3LRzOluWD/d4S0rplZuVfnOPGYA3Nym7fsdux0YmDs998GlsteTe/bnMeuc537Wd3f087/p9jLz+o6o6fXpJfq5dCGQAgB55JitvSfX7k6rFSq/YpJZf5MJZeklnWzYbk+gIXbnRr/G5axz78QGHUnB3+c52+dbWXkNbNj+S977Rmvxtx0jOnvsjux8TnTBMladPU7S2uqw/Z2Z9k5r/+29qvu0Vpd/MLX9TeeJkqSr3iXSryC8JFI/kbnfti3fdjnQeVxHNhf2I5UaWl+8KYJuUeWf3KefI2GrFDt79w0yxg2tyywUNwnZrBDIAwKCQem2DMmu2KTZ7nKL7j2CqfQBx91xnjdsWqn3hOimVlacyuQW12zPyVLZju6frWbtjYxLvC1wdnyQfoIs191dfAxkr+QEAQhU/PPeJXww8ZqbKDx2oyg8duNdj3fNTsu2ZXHDLhzRPdQlsGVdk/xFlXeNvsCCQAQCAopmZFIsOmOnlwYbV4AAAAEJGIAMAAAgZgQwAACBkoQcyMzvdzN4wsxVmdlXY9QAAAJRbqIHMzKKSfijpDEkzJZ1vZjPDrAkAAKDcwh4hmytphbuvcvd2SfdKOivkmgAAAMoq7EA2UdI7Xe6/m9+3GzObb2YLzGxBY2Pjng8DAAAMamEHsj5x91vcvd7d6+vqwm10CgAAELSwA9laSQd0ub9/fh8AAMA+I+xA9ldJ081siplVSDpP0m9CrgkAAKCsQm8ubmYfk3SjpKik2939m3s5vlHS6hKXVStpY4lfY1/G+S0dzm1pcX5Lh3NbWpzf0tnbuT3Q3fd6vVXogWwgMrMFfenMjv7h/JYO57a0OL+lw7ktLc5v6QR1bsOesgQAANjnEcgAAABCRiDr3i1hFzDEcX5Lh3NbWpzf0uHclhbnt3QCObdcQwYAABAyRsgAAABCRiDbg5mdbmZvmNkKM7sq7HqGEjN728xeM7NFZrYg7HoGOzO73cwazGxxl31jzOxxM1uevx0dZo2DVQ/n9hozW5t//y7KL9mDfjCzA8zsj2b2upktMbPL8/t5/xapl3PL+zcAZlZlZi+Z2d/y5/cb+f1TzOzFfHb4RX5t1cKemynLTmYWlfSmpFOV66v5V0nnu/vroRY2RJjZ25Lq3Z21cAJgZn8naYek/3b3Wfl935G02d2/nf8PxWh3vzLMOgejHs7tNZJ2uPt3w6xtKDCzCZImuPtCMxsu6WVJZ0u6SLx/i9LLuf20eP8WzcxMUrW77zCzuKTnJF0u6cuSHnD3e83sJ5L+5u4/LuS5GSHb3VxJK9x9lbu3S7pX0lkh1wR0y92flbR5j91nSbozv32ncv8Qo0A9nFsExN3XufvC/HaTpKWSJor3b9F6ObcIgOfsyN+N579c0ocl/Sq/v1/vXQLZ7iZKeqfL/XfFGzlILukxM3vZzOaHXcwQNc7d1+W310saF2YxQ9BlZvZqfkqT6bQAmNlkSUdKelG8fwO1x7mVeP8GwsyiZrZIUoOkxyWtlLTV3dP5Q/qVHQhkKKcT3P0oSWdI+mJ+Wggl4rnrEbgmITg/ljRV0hxJ6yRdH245g5+ZDZN0v6Qvufv2ro/x/i1ON+eW929A3D3j7nMk7a/czNohQTwvgWx3ayUd0OX+/vl9CIC7r83fNkj6tXJvZARrQ/4akl3XkjSEXM+Q4e4b8v8QZyX9TLx/i5K//uZ+SXe5+wP53bx/A9DdueX9Gzx33yrpj5KOlTTKzGL5h/qVHQhku/urpOn5T0tUSDpP0m9CrmlIMLPq/AWmMrNqSR+VtLj370I//EbShfntCyU9FGItQ8quoJB3jnj/9lv+wujbJC119+91eYj3b5F6Ore8f4NhZnVmNiq/nVDuQ4BLlQtm/5A/rF/vXT5luYf8R4FvlBSVdLu7fzPkkoYEMztIuVExSYpJuptzWxwzu0fSSZJqJW2Q9HVJD0q6T9IkSaslfdrduTi9QD2c25OUm+5xSW9LuqTL9U4ogJmdIOlPkl6TlM3vvlq5a514/xahl3N7vnj/Fs3MZit30X5UuUGt+9z9P/N/4+6VNEbSK5IucPe2gp6bQAYAABAupiwBAABCRiADAAAIGYEMAAAgZAQyAACAkBHIAAAAQkYgAzBkmdkMM1tkZk1m9m9h1wMAPYnt/RAAGLT+XdIf821OAGDAYoQMwFB2oKQl3T1gZtEy1wIAPSKQARiSzOwpSSdL+oGZ7TCzu83sx2b2ezPbKelkM6s0s++a2Roz22BmP8m3Q9n1HF81s3Vm9p6Z/ZOZuZlNC+2HAjBkEcgADEnu/mHlWshc5u7DJLVL+oykb0oaLuk5Sd+WdLByLWWmSZoo6X9LkpmdLul/KterbrqkU8r8IwDYhxDIAOxLHnL35909K6lN0nxJV7j7ZndvknStpPPyx35a0n+5+2J33ynpmlAqBrBP4KJ+APuSd7ps10lKSnrZzHbtM+WaBkvSfpJe7nL86pJXB2CfRSADsC/xLtsbJbVIOszd13Zz7DpJB3S5P6mUhQHYtzFlCWCflJ+2/JmkG8xsrCSZ2UQzOy1/yH2SLjKzmWaWlPT1kEoFsA8gkAHYl10paYWkv5jZdklPSJohSe7+B0k3Snoqf8xTYRUJYOgzd9/7UQAAmZlLmu7uK8KuBcDQwggZAABAyAhkAAAAIWPKEgAAIGSMkAEAAISMQAYAABAyAhkAAEDICGQAAAAhI5ABAACEjEAGAAAQsv8PkzlijJNq7RMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot(loss_list):\n",
    "    plt.figure(figsize=(10,5))\n",
    "    \n",
    "    freqs = [i for i in range(len(loss_list))]\n",
    "    # 绘制训练损失变化曲线\n",
    "    plt.plot(freqs, loss_list, color='#e4007f', label=\"Train loss\")\n",
    "    \n",
    "    # 绘制坐标轴和图例\n",
    "    plt.ylabel(\"loss\", fontsize='large')\n",
    "    plt.xlabel(\"freq\", fontsize='large')\n",
    "    plt.legend(loc='upper right', fontsize='x-large')\n",
    "    \n",
    "    plt.show()\n",
    "\n",
    "plot(loss_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "比较经典全连接神经网络和卷积神经网络的损失变化，可以发现卷积神经网络的损失值下降更快，且最终的损失值更小。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
